Skip to content

Pipe: Fix TsFile reference races and improve result set diffs - #18376

Merged
jt2594838 merged 4 commits into
apache:masterfrom
Caideyipi:fix/pipe-tsfile-reference-race
Aug 31, 2026
Merged

jt2594838 merged 4 commits into
apache:masterfrom
Caideyipi:fix/pipe-tsfile-reference-race

Conversation

@Caideyipi

@Caideyipi Caideyipi commented Jul 31, 2026 •

Copy link
Copy Markdown
Collaborator

Description

PipeTsFileResourceManager.increaseFileReference checked whether a resource existed before entering the creation critical section. Historical and realtime extractors could both observe a missing resource, then create and put separate PipeTsFileResource instances for the same pipe file. The later put overwrote the first reference count, so releasing one event could delete a file that another event still needed. In the reported table pattern IT, this stopped realtime transfer and left the t0 row missing.

This PR:

  • rechecks the private resource map after acquiring the segment lock and reuses the resource created by another caller;
  • covers both hard-linked TsFiles and copied sidecar files such as .mods/.resource;
  • rolls back a private reference when creating the shared/public reference fails, preventing a leaked private reference and file;
  • makes integration-test Set<String> assertions print concise missing and unexpected rows instead of two complete, highly repetitive sets (up to 20 rows per side).

The concurrent regression tests start 64 retain operations together and verify that both private and shared reference counts reach 64, then verify complete cleanup after all releases. A separate failure-path test verifies that a failed public reference increase leaves neither a private count nor a copied file behind.

Validation

  • mvn -pl iotdb-core/datanode -Dtest=PipeTsFileResourceManagerTest test (5 tests passed)
  • mvn -pl integration-test -P with-integration-tests -DskipTests test-compile
  • DataNode Checkstyle and Spotless checks
  • Integration-test Checkstyle and Spotless checks
  • git diff --check

This PR has:

  • been self-reviewed.
    • concurrent write
  • added comments explaining the why and intent where needed.
  • added unit tests for concurrent hard-link and copied-file acquisition.
  • added a unit test for partial-reference failure rollback.

Key changed/added classes
  • PipeTsFileResourceManager
  • PipeTsFileResourceManagerTest
  • TestUtils

@Caideyipi
Caideyipi force-pushed the fix/pipe-tsfile-reference-race branch from ea2ab0e to 25057ef Compare August 5, 2026 07:42
@Caideyipi Caideyipi changed the title Pipe: Fix concurrent TsFile reference increases Pipe: Fix TsFile reference races and improve result set diffs Aug 5, 2026
@jt2594838
jt2594838 merged commit 0116168 into apache:master Aug 31, 2026
36 of 37 checks passed
@jt2594838
jt2594838 deleted the fix/pipe-tsfile-reference-race branch August 31, 2026 01:36
jt2594838 pushed a commit that referenced this pull request Aug 31, 2026
#18544)

* Pipe: Fix concurrent TsFile reference increases

* Test: show concise result set diffs

* Pipe: Roll back failed TsFile reference increases

* Clarify nullable pipe resource map lookup
VGalaxies pushed a commit to VGalaxies/iotdb that referenced this pull request Oct 26, 2026
…#18376)

* Pipe: Fix concurrent TsFile reference increases

* Test: show concise result set diffs

* Pipe: Roll back failed TsFile reference increases

* Clarify nullable pipe resource map lookup
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants